Skip to content

[PULL REQUEST] Startup Module - Blockgroups, Validation, Debug#211

Open
GregorSchroeder wants to merge 3 commits intomainfrom
209-feature-add-blockgroups-to-mgra-table
Open

[PULL REQUEST] Startup Module - Blockgroups, Validation, Debug#211
GregorSchroeder wants to merge 3 commits intomainfrom
209-feature-add-blockgroups-to-mgra-table

Conversation

@GregorSchroeder
Copy link
Copy Markdown
Contributor

@GregorSchroeder GregorSchroeder commented Mar 27, 2026 •
edited
Loading

Describe this pull request. What changes are being made?

Census blockgroups are added to the [inputs].[mgra] table and inserted along with MGRA data as part of the Startup Module.

The Startup Module is given both Validation and Debug functionality by altering the original sql/insert_mgra.sql SQL statement and replacing it with sql/startup/get_mgra.sql and additional Python logic.

What issues does this pull request address?

closes #209
closes #210

Additional context

See [run_id]=13 in [ws].

The following queries will be run in the production database.

ALTER TABLE [inputs].[mgra] ADD [2010_census_blockgroup] NVARCHAR(12) NULL
ALTER TABLE [inputs].[mgra] ADD [2020_census_blockgroup] NVARCHAR(12) NULL

UPDATE [inputs].[mgra]
SET [2010_census_blockgroup] = [xref].[to_zone]
FROM [inputs].[mgra]
INNER JOIN [GeoAnalyst].[geography].[fn_xref_zones](87) AS [xref]
    ON [mgra].[mgra] = CONVERT(INTEGER, [xref].[from_zone])

UPDATE [inputs].[mgra]
SET [2020_census_blockgroup] = [xref].[to_zone]
FROM [inputs].[mgra]
INNER JOIN [GeoAnalyst].[geography].[fn_xref_zones](90) AS [xref]
    ON [mgra].[mgra] = CONVERT(INTEGER, [xref].[from_zone])

ALTER TABLE [inputs].[mgra] ALTER COLUMN [2010_census_blockgroup] NVARCHAR(12) NOT NULL
ALTER TABLE [inputs].[mgra] ALTER COLUMN [2020_census_blockgroup] NVARCHAR(12) NOT NULL

@GregorSchroeder GregorSchroeder requested a review from Copilot March 27, 2026 22:29
@GregorSchroeder GregorSchroeder self-assigned this Mar 27, 2026
@GregorSchroeder GregorSchroeder added the enhancement New feature or request label Mar 27, 2026
@GregorSchroeder GregorSchroeder linked an issue Mar 27, 2026 that may be closed by this pull request
[FEATURE] Add Blockgroups to MGRA table #209
Open
Copilot AI reviewed Mar 27, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the Startup module to support Census blockgroup geographies in the [inputs].[mgra] table and adds validation + debug behavior by splitting the former “insert-only” SQL into a “get/insert switch” query plus Python orchestration.

Changes:

  • Add [2010_census_blockgroup] and [2020_census_blockgroup] columns to [inputs].[mgra] schema definition.
  • Replace sql/insert_mgra.sql with sql/startup/get_mgra.sql, supporting both “return tabular data” and “insert with geometry” modes.
  • Update python/startup.py to fetch MGRA data into pandas for validation and to write debug CSV output.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File Description
sql/startup/get_mgra.sql New Startup MGRA loader query with @insert_switch behavior and added blockgroup xrefs
sql/insert_mgra.sql Removed legacy single-purpose insert script
sql/create_objects.sql Updates [inputs].[mgra] DDL to include the new blockgroup columns (plus added GO separators)
python/startup.py Adds Python-side validation and debug CSV output; switches to new SQL loader

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Eric-Liu-SANDAG
Eric-Liu-SANDAG requested changes Apr 1, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@Eric-Liu-SANDAG Eric-Liu-SANDAG left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to update wiki files

sql/startup/get_mgra.sql
Copy link
Copy Markdown
Contributor

@Eric-Liu-SANDAG Eric-Liu-SANDAG Apr 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • [xref_id]=87 in [GeoAnalyst] says "Many-to-many cross-reference between Series 15 MGRAs and 2010 Census Block Groups..." instead of "Many-to-many cross-reference between MGRA15 points and 2010 Census Block Groups..."
  • It is necessary to document why we sometimes use point based vs area based xrefs? Maybe something to add to the wiki?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

@Eric-Liu-SANDAG Eric-Liu-SANDAG Eric-Liu-SANDAG requested changes

Requested changes must be addressed to merge this pull request.

Assignees

@GregorSchroeder GregorSchroeder

Labels

enhancement New feature or request

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[FEATURE] Startup Module Validate and Debug Functionality [FEATURE] Add Blockgroups to MGRA table

3 participants

@GregorSchroeder @Eric-Liu-SANDAG